Method and system for luminance noise filtering

ABSTRACT

In a method and system for luminance noise filtering, a region of pixel data directly from the image sensor is used for determining a virtually filtered luminance for a pixel location within the region. Luminance noise reduction is performed using the region of pixel data directly from the image sensor such that frame memory is eliminated. In addition, the present invention provides adaptive noise filtering by selecting the virtually filtered luminance as a final luminance for a darker image and by selecting a reference luminance without virtual noise filtering for a brighter image.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present application is a continuation application of an earlierfiled copending patent application with Ser. No. 10/776,447 filed onFeb. 10, 2004, for which priority is claimed. This earlier filedcopending patent application with Ser. No. 10/776,447 is in its entiretyincorporated herewith by reference.

The present application also claims priority under 35 USC § 119 toKorean Patent Application No. 2003-0037268, filed on Jun. 10, 2003, inthe Korean Intellectual Property Office, the disclosure of which isincorporated herein in its entirety by reference. A certified copy ofKorean Patent Application No. 2003-0037268 is contained in the parentcopending patent application with Ser. No. 10/776,447.

TECHNICAL FIELD

The present invention relates generally to image pick-up devices, andmore particularly, to luminance noise filtering for a pixel locationusing a region of pixel data directly from an image sensor such thatline memory capacity is minimized.

BACKGROUND OF THE INVENTION

FIG. 1 illustrates an image pick-up device 102 such as a camera systemthat includes an image sensor 104. The image sensor 104 generates pixeldata for an image of an object 106 that is projected through anobjective lens 108 onto the image sensor 104. For example, the imagesensor 104 may be a CIS (CMOS image sensor) commonly used in hand-helddevices such as cell phones and PDA's (personal digital assistants).

A signal processor 110 manipulates the pixel data from the image sensor104 for showing the image of the object 106 on a display 112, or forfurther processing by an image recognition system 114, or for sendingthe image via a transmission system 116 such that the image is shown ona remote display 118. Referring to FIGS. 1 and 2, the image sensor 104generates pixel data 120 according to a Bayer filter array overlying theimage sensor 104.

With the Bayer filter array, the image sensor 104 generates an intensitysignal of a respective color at each pixel location. A square labeledwith an “R” is for a pixel location on the image sensor 104 thatgenerates an intensity signal of the red color component. Similarly, asquare labeled with a “G” is for a pixel location on the image sensor104 that generates an intensity signal of the green color component.Further, a square labeled with a “B” is for a pixel location on theimage sensor 104 that generates an intensity signal of the blue colorcomponent.

An interpolation algorithm is then used by the signal processor 110 todetermine the full set of intensity signals of the respectiveinterpolated RGB color components for each of the pixel locations. Theinterpolation algorithm uses the pixel data of the Bayer color filterarray 120 for such a determination.

Such an interpolation algorithm is known to one of ordinary skill in theart as disclosed in U.S. Pat. No. 5,382,976, U.S. Pat. No. 5,506,619, orU.S. Pat. No. 6,091,862. For determining the interpolated colorcomponents R′, G′, and B′ at a particular pixel location 124 with suchan interpolation algorithm, a region of pixel data 126 surrounding thatpixel location 124 is used as illustrated in FIG. 3.

Temporal noise affects the quality of the image of the object asdetected and generated by the image pick-up device 102. Temporal noiseis the variation in the output from the image sensor 104 even underuniform illumination onto the image sensor 104. Such temporal noise mayarise from shot noise and 1/f noise at the photo-diodes of the imagesensor 104, from thermal noise at the transistors and other circuitcomponents used within the image sensor 104, or from quantization errorof an A/D (analog to digital) converter used within the image sensor104.

Such temporal noise increases with brightness of the image. However, thedetrimental effect of the temporal noise on the image is greater atlower illumination because the SNR (signal to noise ratio) decreaseswith lower illumination. In fact, temporal noise sets a limit on thedynamic range of the image sensor 104 under dark conditions.

FIG. 4 illustrates a prior art process for reducing the effect of suchtemporal noise. The pixel data 120 is generated with the Bayer colorfilter array at the image sensor 104. The signal processor 110interpolates such pixel data 120 to generate the respective interpolatedRGB color components 122A, 122B, and 122C that are stored within a framememory device 122 of the prior art.

In the prior art, after the interpolated RGB color components 122A,122B, and 122C for an n×n array of pixel locations are generated andstored in the frame memory device 122, a noise reducing block 132 usessuch interpolated RGB color components for reducing the deleteriouseffects of the temporal noise. FIG. 4 shows the noise reducing block 132using the 3×3 arrays of the interpolated RGB color components 122A,122B, and 122C. However, other prior art noise reducing processes mayalso use 5×5, 7×7, or other n×n arrays of the interpolated RGB colorcomponents.

In any case for the prior art noise reducing process, the capacity ofthe frame memory device 122 is sufficient to store the n×n arrays ofinterpolated RGB color components used by the noise reducing block 132.However, such a relatively large capacity of the frame memory device 122is disadvantageous when the camera system 102 is incorporated as part ofa hand-held device such as a cell phone or a PDA for example. Thus,elimination of the frame memory device 122 is desired for a smallerdevice size, lower power dissipation, and lower cost especially when thecamera system 102 is incorporated into a hand-held device.

SUMMARY OF THE INVENTION

Accordingly, in a general aspect of the present invention, luminancenoise filtering is performed for a pixel location using a relativelysmall region of pixel data directly from the image sensor such that theframe memory may be eliminated.

In a general embodiment of the present invention, in a method and systemfor luminance noise filtering, a region of pixel data from the imagesensor is used for determining a virtually filtered luminance for apixel location within the region. In an example embodiment, thevirtually filtered luminance is determined by averaging the respectivepixel data multiplied with a respective weighting coefficient for eachpixel location of the region.

In another embodiment of the present invention, the color components forthe pixel location are determined from the region of pixel data.

In a further embodiment of the present invention, a reference luminanceis determined for the pixel location from the color components. A finalluminance of the pixel location is selected between the virtuallyfiltered luminance and the reference luminance depending on an adaptiveluminance that indicates the brightness of the image. The presentinvention provides adaptive noise filtering by selecting the virtuallyfiltered luminance as the final luminance for a darker image.

In this manner, noise filtering is performed for a pixel location withina region of pixel data using virtual luminance that is determined usingsuch a region of pixel data directly from the image sensor. Thus, framememory for storing interpolated pixel data is eliminated with thepresent invention. The elimination of frame memory is especiallyadvantageous for smaller device size, lower power dissipation, and lowercost of the camera system incorporated into a hand-held device.

These and other features and advantages of the present invention will bebetter understood by considering the following detailed description ofthe invention which is presented with the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows components of an image pick-up device such as a camerasystem, according to the prior art;

FIG. 2 illustrates interpolation of pixel data generated with a Bayercolor filter array into color components, according to the prior art;

FIG. 3 illustrates use of a region of pixel data generated with a Bayercolor filter array for determining the color components of a pixellocation within the region, according to the prior art;

FIG. 4 illustrates a process for performing noise reduction after a n×narray of color components has been determined, according to the priorart;

FIG. 5 illustrates a process for performing noise filtering using aregion of pixel data for determining the luminance and the colorcomponents of a pixel location, according to an example embodiment ofthe present invention;

FIG. 6 shows a block diagram of a system for performing noise filteringas illustrated in FIG. 5 such that frame memory is eliminated, accordingto an example embodiment of the present invention;

FIG. 7 shows a flowchart during operation of the system of FIG. 6,according to an example embodiment of the present invention;

FIG. 8 illustrates use of a region of pixel data generated with theBayer color filter array for determining color components of a pixellocation within the region in a step of the flowchart of FIG. 7,according to an example embodiment of the present invention;

FIGS. 9A, 9B, and 9C illustrate use of the region of pixel data of FIG.8 for determining a plurality of virtual luminance arrays, according toan example embodiment of the present invention;

FIG. 10 shows a block diagram of a noise filter within the system ofFIG. 6, according to an example embodiment of the present invention;

FIG. 11 shows an example graph used by a data processor within thesystem of FIG. 6 for determining a threshold value depending on anadaptive luminance, according to an example embodiment of the presentinvention; and

FIG. 12 shows another example graph used by the data processor withinthe system of FIG. 6 for determining the threshold value depending on anauto exposure gain of the camera system, according to an exampleembodiment of the present invention.

The figures referred to herein are drawn for clarity of illustration andare not necessarily drawn to scale. Elements having the same referencenumber in FIGS. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12 refer toelements having similar structure and function.

DETAILED DESCRIPTION

Referring to FIGS. 5 and 6, a system 200 of a general aspect of thepresent invention performs noise filtering using a region of pixel datadirectly from the image sensor 104. In addition, the noise filtering isperformed during determination of the luminance using such a region ofpixel data for a single pixel location within the region. FIG. 7 shows aflowchart of steps during operation of the system 200 of FIG. 6.

The system 200 is typically implemented within the signal processor 110for manipulating pixel data from the image sensor 104. Referring toFIGS. 6, 7, and 8, a RGB matrix 202 includes a line memory device 204for inputting and storing a region of pixel data 206 generated with aBayer color filter array at the image sensor 104 (step 302 of FIG. 7).

The line memory device 204 may be implemented with any type of datastorage devices. For example, the line memory device 204 may store pixeldata for 4×H pixel locations with H being the number of columns ofpixels at the image sensor 104 (such as 648 columns for example) whenthe image sensor 104 outputs pixel data row by row.

The RGB matrix 202 also includes an interpolation processor 208 fordetermining interpolated color components R′, B′, and G′ of a pixellocation 210 within the region 206 (step 304 of FIG. 7). Referring toFIG. 8, the interpolation processor 208 interpolates the region of pixeldata 206 to generate the interpolated color components R′, B′, and G′.Such interpolation algorithms for generating the interpolated colorcomponents R′, B′, and G′ are known to one of ordinary skill in the art.

In addition, the RGB matrix 202 determines a reference luminance, Y1H,from the interpolated color components R′, B′, and G′ as follows (step304 of FIG. 7):

Y1H=(19*R′+38*G′+7*B′)/64

Such a reference luminance Y1H is calculated according to a conventionalstandard in the industry for calculating luminance as known to one ofordinary skill in the art.

Furthermore, the RGB matrix 202 includes a virtual luminance processor212 for determining virtual luminance arrays Y0V, Y1V, and Y2V (step 304of FIG. 7). Referring to FIG. 9A, a first virtual luminance array Y0V=[AB C], with A, B, and C each being a respective average of the intensityvalues of a respective four pixels included in a box correspondinglylabeled as A, B, and C in the region of pixel data 206.

Similarly, referring to FIG. 9B, a second virtual luminance array Y1V=[DE F], with D, E, and F each being a respective average of the intensityvalues of a respective four pixels included in a box correspondinglylabeled as D, E, and F in the region of pixel data 206. Also, referringto FIG. 9C, a third virtual luminance array Y2V=[G H I], with G, H, andI each being a respective average of the intensity values of arespective four pixels included in a box correspondingly labeled as G,H, and I in the region of pixel data 206. In this manner, the virtualluminance arrays Y0V, Y1V, and Y2V are determined for the pixel location210 using the region of pixel data 206 that also determines theinterpolated color components R′, G′, and B′ for the pixel location 210.

Referring back to FIG. 6, the RGB matrix 202 sends the referenceluminance Y1H and the virtual luminance arrays Y0V, Y1V, and Y2V to anoise filter 214. In addition, the RGB matrix 202 sends the interpolatedcolor components R′, G′, and B′ to a chrominance signal processor 215within a Y/C (luminance/chrominance) processor 216. Furthermore, the RGBmatrix 202 sends the reference luminance Y1H and the interpolated colorcomponents R′, G′, and B′ to a data processor 218.

FIG. 10 shows a block diagram for an example embodiment of the noisefilter 214 that determines a virtually filtered luminance GOUT (step 306of FIG. 7) from the virtual luminance arrays Y0V, Y1V, and Y2V. Thenoise filter 214 includes multipliers 222, 224, and 226 and an adder 228within a multiplying and summing block 230.

The first multiplier 222 multiplies the first luminance array Y0V with afirst weighted coefficient array GAD0 [39 63 39] as follows:

39*A+63*B+39*C.

Similarly, the second multiplier 224 multiplies the second luminancearray Y1V with a second weighted coefficient array GAD1 [63 104 63] asfollows:

63*D+104*E+63*F.

Also, the third multiplier 226 multiplies the third luminance array Y2Vwith a third weighted coefficient array GAD2 [39 63 39] as follows:

39*G+63*H+39*I.

The adder 228 sums together the resulting values from the multipliers222, 224, and 226. A shifter 232 within a brightness control block 234divides the result from the adder 228 with a sum of all the coefficientsof GAD0, GAD1, and GAD2 (i.e., 512). A fourth multiplier 236 within thebrightness control block 234 multiplies the result from the shifter 232by a luminance compensation factor α, to generate the virtually filteredluminance GOUT.

In one embodiment of the present invention, the luminance compensationfactor α and the weighted coefficient arrays GAD0, GAD1, and GAD2 aredetermined using a Gaussian distribution equation for optimum imagequality and are stored within a data register 220 in FIG. 6. In thismanner, the virtually filtered luminance GOUT is determined by averaginga respective pixel data multiplied with a respective weightingcoefficient for each pixel location of the region 206.

In addition, referring to FIG. 6, the data processor 218 determines athreshold value THV (step 306 of FIG. 7). Referring to FIG. 11, the dataprocessor 218 determines THV dependent on an adaptive luminance, AY, asillustrated by the graph 223 of FIG. 11. The adaptive luminance, AY,indicates the overall brightness of a previous image, in one exampleembodiment of the present invention.

For example, in FIG. 12, THV is determined according to another graph225 dependent on the auto exposure gain used by the camera system 102.The auto exposure gain indicates a brightness of the image to bedetected by the image sensor 102 within the camera system 102, as knownto one of ordinary skill in the art. A higher auto exposure gainindicates a darker image for a higher THV according to the graph 225 ofFIG. 12. In such an example, the auto exposure gain indicates theadaptive luminance AY.

In another embodiment of the present invention, the reference luminanceY1H is used as indicating the adaptive luminance AY in FIG. 11. In thatcase, THV is determined depending on a brightness level for each pixellocation. Alternatively, the adaptive luminance AY is an average of therespective reference luminance Y1H for a predetermined region of pixeldata. In any case, the THV is determined to be higher for a darker imageaccording to the graph 223 of FIG. 11.

Referring back to FIG. 10, a comparing block 238 includes a subtractor240 and an absolute value generator 242 that determine an absolute ofthe difference between the virtually filtered luminance GOUT and thereference luminance Y1H (i.e., delta) (step 308 of FIG. 7). A comparator244 within the comparing block 238 compares delta with THV (step 310 inFIG. 7).

In FIG. 10, a multiplexer 246 inputs the reference luminance Y1H and thevirtually filtered luminance GOUT. The comparator 244 controls themultiplexer 246 to select and output the virtually filtered luminanceGOUT as a final luminance of the pixel location 210 when delta is lessthan or equal to THV (step 312 of FIG. 7). Alternatively, the comparator244 controls the multiplexer 246 to select and output the referenceluminance Y1H as the final luminance of the pixel location 210 whendelta is greater than THV (step 314 of FIG. 7).

If the pixel location 210 is a last pixel location for an image (step316 of FIG. 7), the flowchart of FIG. 7 ends. Otherwise, steps 302, 304,306, 308, 310, 312, 314, and 316 are repeated for another pixel locationwith another region of pixel data from the image sensor 104. Such stepsare repeated for determining the respective luminance and interpolatedcolor components for each pixel location with a respective region ofpixel data of the image.

Generally, a larger array of pixel data is generated from the imagesensor 104 for producing a smaller array of the processed image, asknown to one of ordinary skill in the art. Pixel data from locationstoward the outer perimeter of the image sensor 104 are used for imagesignal processing of adjacent pixel locations toward the center of theprocessed image. However, such pixel locations toward the outerperimeter become cut off from the processed image because a sufficientregion of pixel data surrounding such a pixel location is not available,as known to one of ordinary skill in the art. Steps 304, 306, 308, 310,312, 314, and 316 of FIG. 7 are repeated for each of the pixel locationsfor the processed image.

In this manner, noise filtering is performed for a single pixel locationusing a region of pixel data 206 directly from the image sensor 104while determining the final luminance Y1H/GOUT for the pixel location210. The region of pixel data 206 is also used for determining theinterpolated color components R′, G′, and B′ of the pixel location 210.Because noise filtering is performed by using the region of pixel data206 directly from the image sensor 104, frame memory for storinginterpolated pixel data may be eliminated with the present invention.

Thus, the capacity of the memory device included in the image pick-updevice may be minimized since the luminance noise filter uses the imagedata directly from the image sensor for determining the final luminanceY1H/GOUT and the interpolated color components R′, G′, and B′ of thepixel location 210. Such smaller memory capacity is advantageous forsmaller device size, lower power dissipation, and lower cost especiallywhen the camera system is incorporated into a hand-held device.

In addition, the present invention provides adaptive noise filtering byvarying the threshold value THV depending on the brightness of theimage. For a brighter image, the reference luminance Y1H is selected asthe final luminance instead of the virtually filtered luminance GOUT.Noise filtering introduces distortion to the image, and the effect oftemporal noise is less for a brighter image. Thus, the referenceluminance Y1H without distortion from virtual noise filtering isselected as the final luminance for a brighter image. On the other hand,the deleterious effect of temporal noise is greater for a darker image.Thus, the virtually filtered luminance GOUT with noise filtering isselected as the final luminance for a darker image.

Referring back to FIG. 6, a luminance signal processor 217 within theY/C processor 216 processes the final luminance Y1H/GOUT for performingcontour compensation to output a luminance signal Y for a Y/C(luminance/chrominance) formatter 219. A chrominance signal processor217 within the Y/C processor 216 receives the interpolated colorcomponents R′, G′, and B′ and auto white balance data (AWBD) from thedata processor 218 to generate further color components R″-Y and B″-Yfor the Y/C formatter 219. The R″ is an intensity of the interpolatedcolor component R′ which is adjusted by the auto white balance data, andthe B″ is an intensity of the interpolated color component B′ which isadjusted by the auto white balance data.

The Y/C formatter 219 generates luminance and chrominance data Y, Cb,Cr/R, and color data R′″, G′″, and B′″ according to a standard asrequired by the display 112, the image recognition system 116, or thetransmission system 116. Such components of the Y/C processor 216 andthe Y/C formatter 219 are known to one of ordinary skill in the art. Thedata processor 218 also determines and outputs to the image pick-updevice auto exposure control data AED from Y1H, R′, G′, and B′, as knownto one of ordinary skill in the art. The present invention lies in thedarkly outlined components of the RGB matrix 202, the noise filter 214,the data processor 218, and the data register 220 in FIG. 6, and theother components of FIG. 6 are known to one of ordinary skill in theart.

The foregoing is by way of example only and is not intended to belimiting. For example, the present invention is described for the camerasystem 102 that may be part of a hand-held device. However, the presentinvention may be used for any type of imaging device performing imagesignal processing. In addition, the components illustrated and describedherein for an example embodiment of the present invention may beimplemented with any combination of hardware and/or software and indiscrete and/or integrated circuits. In addition, any number asillustrated and described herein is by way of example only. For example,any number of pixels as illustrated and described herein is by way ofexample only.

The present invention is limited only as defined in the following claimsand equivalents thereof.

1. A method for luminance noise filtering, comprising: inputting aregion of pixel data from an image sensor; determining a virtuallyfiltered luminance from a first processing of said region of pixel dataand without using other pixel data for a pixel location within theregion; and determining a reference luminance for the pixel locationfrom a second processing of said same region of pixel data and withoutusing other pixel data, wherein the reference luminance is determinedafter respective interpolated color components for the pixel locationare determined such that the reference luminance is determined usingsaid respective interpolated color components.
 2. The method of claim 1,wherein the second processing includes the steps of: determining saidinterpolated color components for the pixel location from said region ofpixel data; and determining the reference luminance for the pixellocation from the interpolated color components.
 3. The method of claim1, further comprising: selecting between the virtually filteredluminance and the reference luminance as a final luminance of the pixellocation depending on an adaptive luminance.
 4. The method of claim 3,further comprising: determining a threshold value from the adaptiveluminance; selecting the virtually filtered luminance if an absolute ofa difference between the virtually filtered luminance and the referenceluminance is less than or equal to the threshold value; and selectingthe reference luminance if the absolute of the difference between thevirtually filtered luminance and the reference luminance is greater thanthe threshold value.
 5. The method of claim 4, wherein the adaptiveluminance is determined from an average reference luminance for apredetermined region of pixel data.
 6. The method of claim 3, whereinthe adaptive luminance is indicated by an auto exposure gain for theimage sensor.
 7. The method of claim 3, wherein the adaptive luminanceis indicated by the reference luminance.
 8. The method of claim 1,wherein the virtually filtered luminance is determined by averaging arespective pixel data multiplied with a respective weighting coefficientfor each pixel location of the region.
 9. The method of claim 1, whereinthe image sensor is part of a hand-held image pick-up device havingminimized line memory capacity.
 10. A system for luminance noisefiltering, comprising: a memory device for storing a region of pixeldata from an image sensor; a noise filter for determining a virtuallyfiltered luminance from a first processing of said region of pixel dataand without using other pixel data for a pixel location within theregion; and a matrix for determining a reference luminance for the pixellocation from a second processing of said same region of pixel data andwithout using other pixel data, wherein the matrix determines thereference luminance after respective interpolated color components forthe pixel location are determined such that the reference luminance isdetermined using said respective interpolated color components.
 11. Thesystem of claim 10, wherein the matrix determines said interpolatedcolor components for the pixel location from said region of pixel datasuch that the reference luminance is determined from the interpolatedcolor components.
 12. The system of claim 10, wherein the noise filterselects between the virtually filtered luminance and the referenceluminance as a final luminance of the pixel location depending on anadaptive luminance.
 13. The system of claim 12, further comprising: adata processor that determines a threshold value from the adaptiveluminance; wherein the noise filter selects the virtually filteredluminance if an absolute of a difference between the virtually filteredluminance and the reference luminance is less than or equal to thethreshold value; and wherein the noise filter selects the referenceluminance if the absolute of the difference between the virtuallyfiltered luminance and the reference luminance is greater than thethreshold value.
 14. The system of claim 13, wherein the adaptiveluminance is determined from an average reference luminance for apredetermined region of pixel data.
 15. The system of claim 12, whereinthe adaptive luminance is indicated by an auto exposure gain for theimage sensor.
 16. The system of claim 12, wherein the adaptive luminanceis indicated by the reference luminance.
 17. The system of claim 10,wherein the virtual luminance is determined by averaging a respectivepixel data multiplied with a respective weighting coefficient for eachpixel location of the region.
 18. The system of claim 10, wherein theimage sensor is part of a hand-held image pick-up device havingminimized line memory capacity.
 19. A system for luminance noisefiltering, comprising: means for inputting a region of pixel data froman image sensor; means for determining a virtually filtered luminancefrom said region of pixel data and without using other pixel data for apixel location within the region; and means for determining a referenceluminance for the pixel location from a second processing of said sameregion of pixel data and without using other pixel data, wherein thereference luminance is determined after respective interpolated colorcomponents for the pixel location are determined such that the referenceluminance is determined using said respective interpolated colorcomponents.
 20. The system of claim 19, further comprising: means fordetermining said interpolated color components for the pixel locationfrom the said region of pixel data; and means for determining thereference luminance for the pixel location from the interpolated colorcomponents.
 21. The system of claim 20, further comprising: means forselecting between the virtually filtered luminance and the referenceluminance as a final luminance of the pixel location depending on anadaptive luminance.